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ABSTRACT 

^ General directions in evaluating U.rS. colleges and 

universities and problems that might be encountered in evaluating 
institutional quality are discussed. Eight guidelines for assessing 
institutional quality are also offered^ Appendices consider the 
f ollowihg five in sti tut ibhal_ evaluation apprbacBesi educational- _ 
audits^ external consultant reviews^-selfr 

self ^studies f or-insti^utional purposes^ and state and federal _ 
reviews, included are a categorization of statements 
for an institutional evaluation that cover goals and objectives, 
student learning, faculty performance, academic programs^ 
institutional support services^ administrative leadership^ financial 
manigement, governing boards external relations, and institutional 
self--imprbvemeht. Suggestion institutional quality, 

include : making- theoret icalan^ 

evaluation- compatible with the organizational ethos; commtxni eating an 
overall plan for enhancing xnstitutionaiq^^ linking models and 

plans for evaluating institutional quality with resource allocation; 
and making institutional evaluation studies action-oriented, with 
plans for moving reports-to-action. (SW) 
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EVALUATING INSTITUTIONAL QyALITY; 

SOME Ways and some problems* 



Ins t 1 t a t t ona i evaluation very likely dates back to the early days 
of formal^ Western edacation, and one can iiiia^ihe that a robust 
evai dative dialogue must have taken place in ancient Greece between 
the merits of Isocrate^ and his school of orators and Aristotle and 
his academy. 

In 6 t i t a t i ona i evaluation in the United States can be datel back 
to the early days of Harvard* or even to September 3, 1642. 0«4 that 
day. ^ohn W^nthrbp, Governor of the Massachusetts Bay Company^ 
journeyed from Boston to Cambridge and presided over the public final 
examihatioh given to nine seniors of Harvard College by its external 
Board of Overseers <Harcierod» 1980. p. 1). 

Members of the higher education sector currently have ample 
opportunity to make a decisive impact upon the growing trend toward 
program and/or institutional evaluation if we give sufficient 
attention to meaningful research. If we exercise political savvy, and 
if we oppose creatively yet vigorously those who propose simplistic 
ard misleading solutions to complex problems; In many cases it is 
counterproductive for us to propose further study without action 
expectations. Sometimes the external demands for decisions will hot 
wait very long; 



* Introductory paper by Richard I. Miller, Hi ghe r _ Educa 1 1 oh 

Program, Ohio Uni vers 1 ty , Athens , Ohio, U^S.A.; for the OECD's _ 
Prografnme on Institutional Manageiient In Higher Educat Ion , Special 
^ Tbi>ic Workshop oh "The Role of Eyaluatloh in the Management of 
^ Ihstitutiohs of Higher Educatioh"^ May 12-14^ 1986^ Parls^ France. 

^ i 
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Gehera4^Dlre<:tten<- in Institutional Evaluatibh 
A number of general direccibns in institutional evaiuatton in the 
U.S.A. may be identified. 

J ^ Mo re institutional e valuations wi ll be undertaken, bo th from 
e xternal pressures such as b u s i ne s s / i hdu s t r v . and from state^&nd 
j^e^ral governmental forced . 

The state legislatures in their 1985-86 appropriations for higher 
education have been :r.ore generous than in the previous several years, 
due -argely to the current eiiiphai;is oh quality education, to 
reascnably iood state level and national e. nomies. and tO feelings on 
the part of state officials that higher education has been squeezed 
enough--at least for awhile. 

But there is another side also, that of increasing restiveness on 
the part 8f external forces about accountability, or more precisely, 
our alleged lack of it anong colleges and universities. 1.-5. cblleg;«is 
and universities may be managed better than th. external forces 
believe is the case, but even with this jodgmeat there seems tO he a 
inalaise abSut effective and efficient Management among those in 
academe who do it for a living. This serendipity may be short lived 
due to predictions of declines in student enrollments as well as in 
the economy. With the need for mOre money for operations and fewer 
students in the classrooms, the external pressures fOr accountability 
will become greater. 
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2 ; The United States is experiencinj^ a^ n unpr e ced ented surke of 
teres: in qua lity educ a^ lcn ^ 

Never before in our history has so much atCentioH been concen- 
trated on quality education in repdrtsi television programs, articles, 
conferences and conversations. 

This c i r c uin s t ahce augurs well for iniprbvihg highei education in 
the United States provided educators take advantage of this ground- 
swell of interest by working indre diligently for greater improvements 
in the quality of education. As yet, we see only the beginnings of 
qualitative improvements but we hope that the t-rehds will continue in 
a positive manner. If it is difficult for those who do not live and 
work in the U.S.A. to v^ndefstand the plethora of hatidhal repdrts that 
have been issued to improve our system of higher education^ be my 
guest. I would guess that very few of us fully understand what is 
happening either. But it is obvious that the current surge of reports 
and interests in quality education is far greater than any previous 
one. If only 2D percent of the most prominent recommehdat ions are 
implemented* which is about the efficiency of a gasoline engine, we 
should be quite pleased. 

3 . The very rapid growth in inte^est^ in va?lue-added education and in 
s-t ud e^h^— du t cdm^s^ measurement will continue for a numbe^^ bf years . 

The value-added approach^ simply stated, entails measurentent at 
the beginning of a course ahd/dr program and ahdther measurement at 
its conclusion. The difference is what we call "value-added" because 
df the prdgrara. The term "student outcdmes" refers to what knowledge, 
values, or attitudes that the student has mastered at its conclusicn^ 
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as measured by some kind of test; Most often the oatcomes are 
knowledge or factual information rather than values and attitudei. 

Much discussion and literature in recent years ha; focused oh 
student outcomes; in other words* what does the undergraduate student 
take from a college after four years? In a recent article, Peter Ewell 
wrote: "What should students be expected to know by graduation? Much 
of the difficulty in establishing assessment programs is the general 
tack of agreement oh the answer to this question" (1985, p. 355. 

Some earlier, simplistic approaches have given way to more 
sophistic »ted ones. Student outcotKus can be measured in a number of 
ways, including tests for professional certification or licensure. A 
wide array of these tests cah be found in Burros's Mental H e a s u r e^>^e^vts 
Yearbooks . and the latest edition lists standardized instruments for 

over 40 subject areas. 

The current exteniive program revisions in the general education 
coinpoSeht in U.S. colleges and universities is evidence that testing 
for student ontcoSes should include greater attention to the broad 
aesthetic and cultural aspects as well as preparation for one's 
lifewofk- the American College Testing Program has developed the 
CSileie Outcome Measurements Project (C0MP5 lor assessing general 
education khSwledie ano skills. The oatcomes test covers the six 
themes of communicating, sclving problems, clarifying values, 
functioning within social institutions, using science and technology, 
and using the arts. the ootcoSei assessments at the University of 
tennessee, knSxville and at Northeast Missouri State University are 
making extensive use of COUP test results. 
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I Chambers has written that for outcome measures to be used for 

evaluating educational quality they should meet two tests: "Firsts 
each outcome should represent something that is teachable; that is» it 
is generally accepted that the particular educational objective 
adopted can beachieved through an 1 n s t v uc t i ona 1 cr didactic process. 
. . . Secoridt each outcome should measure something judged relevant to 
the graduate*s functioning throughout adult life" (1979| p. 31). 

Some individuals have confused the value-added approach 
academic stahdardsi and sot^ how have alleged that the valu^*added 
approach will louer acsdenii^ standards* One can make th^ case, quite 
to the contrary^ as Alexander Astin has donei that considering 
acadeitiic standards as the same as value-added standards can provide a 
much more vigorous and far less ambiguous set of standards than 
current academic standards based solely on relativistic course grades** 
(1983* p. 136). 

4 ; The rating game win continue • 

Every year numerous ratings of U.S. colleges and universities 
provide information on the best party schools in the hatibhi campUs »s 
with the most active sports programs » those with the prettiest or most 
handsome students, most beautiful c amp u se s --and other equally 
significant information on our academic institutions. 

Perhaps symbolic of the current popularity of campus ratings is 
the cover story article in the November /December 1984 issue of Change 
magazine, which was: **Who Is Jack GbUrmari: And Vhy Is He Saying All 
Those Things About My College?'* (The first Gburmah report was 
published in 1967 with little fanfar and even less notoriety.) And 
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the cover story article in the November /December 1985 Issue of 
magazine was titled: **Are Colleges Hakihg 'he Grade?" 

To learn more aboat how students select colleges^ The Carnegie 
Foandatibh for the Advauceroeht of Teaching surveyed 1«000 high school 
seniors about the sources of information that they used most 
frequently in deciding on a college. The survey "revealed that nearly 
60 percent of all prospective college students read commercial college 
gtiides. The students and their parents rated the relevancy and 
accuracy of these guides higher than college publications. But trying 
to pick a good guide is almost as complicated as trying to pick a 
college" (1986, pp, 3©-3i). 

The ratings can provide useful information, however. They can 
assist parents and couns e 1 or s . i n sorting through bur bvcr-3000 post- 
secondary institutions of higher education; they can assist graduate 
students in making educatt oral decisions; they can provide ihstitu-- 
tionai yardsticks; and they can provide governing boards with 
comparative information. But they also can provide simplistic and 
sbmetines distorted information about complex problems. ^ 
5. ^^4i_^re becoming rebre prbficieht in unde rtaking i nstt^tionar 
evaluatiohs^ . 

/ : - : : : : : ' 

bar imprbvement over the past 10 years has been due to 

considerable practice in this craft. While practice dbcs hot make 

perfect^ we htkve learned from bur mistakes. Also, we have begun to use 

mbre effectively the technologies at our disposal. The wbrk at the 

National Center fbr Higher Education Management Systems (NCHEMS) has 

prbvided leadership in developing evaluatibn systems. 
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What Kind of I ns 1 1 1 u t f ong 1 -E-va 1 g » t i on^ 
What kind of evaluation can be inbit useful? What Kipling once 
wrdtc--"There are nine and sixty ways of constructing tribal lays, 
and-every-single-bhe-6f-thein-is-right'^ up the importance of 

developii\g an approach to evaluation that is tailored to the specific 
institucion* 

Two general types of evaluation can be identified. One type 
consists of those evaluations that are less than ihstitutibh-wide in 
scope, sach as the assessments of academic programs, faculty^ 
studehtsi and other areas. However, a systematic and extensive plan 
for academic program evaluation may become r.ore closely akin to the 
second type, which is institution-wide evaluation. 

The second type of appraisal, that of the entire college or 
university, has received less attention, although interest in this 
dimension now is growing substantially ir. the U.S.A. One can identify 
at least five approaches to Institutional evaluation. These ar« i. 
educational auditing^ 2. external consultant reviews* 3. self-studies 
for accreditation, 4. self-studies for institutional purposes, and 5^ 
state and federal reviews. A brief discussion of these five approaches 
is Included as an appendix. 

General Guidelines for Institutional Eval4j at ibnr 
If you are considering or planning to undertake a major study of 
institutional quality^ what are some guidelines that yoii may want to 
consider? Eight guidelines are sugges'ced for your consideration: 
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1 . Thebfettca l and operational plans for evaiaatt4vg 1 n s 1 1 1 U t id^^a 1 
quality shbald be^ompa t i b 1 e ^i xh the or r ah i t i bna 1 ethoc. 

Simply put^ one should abide by the classic anthtopolbgical 
concept of /'Adapt not adorstc" the arguitient for "adoption" is that the 
internal structure and dyhamisn of a plan is less likely to be 
destroyed when the whole concept or inodel is adopted. 

On the other hand^ each college or university has a unique 
history and nature that can result in forcing a square into a round 
hole if pure adoption is pursued, the U.S.A. experience leans strongly 
toward modifying and tailoring concepts and platis to fit the 
circumstances and needs of each situation. 

2 • Vigorous yet sensitive admini s t r a t i ve leade r sh i p is essential . 

This leadership includes providing the initial impetus for the 
evaluation^ keeping track of progress and/or problems, and seeing that 
the emphasis is toward some ultimate concrete r ecommehda t ions for 
ac t i on . 

The management of higher education is receiving renewed attehtibh 
as the skills for managing institutional decline have become more 
prevalent in recent years than those needed fbr iiiahagihg i hs t i t u t i bhal 
growth* And the natter is made more complicated because today's senior 
administrative officers have a three ring circus to contend with. In 
ring one is academic programs that have declining student enr o 1 1 men t » ^ 
In ring two are those with stable enrollments, and in ring three are 
programs that are or can be increased in student size. And such 
adjustments often are made In an overall budget that i$ only tiiodestiy 
increasing! or in some cases it is decreasing.. 
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Bat what kind of leadership are we talking about? A debate about 
the most effective styles of academic leadership has been going on for 
some time- the 1974 book by Cohen and March, titled. Amb^gotty and 
Leadership > received wide attention in the U.S. A, Its findings were 
based upon data gathered primarily during the student unrest in the 
lane 196D*s. They described the university and the presidency in this 
manner : 

Decision making in the u n i ve r s 1 t y «e e^s to result 
extensively from a process that decoupl e s problems and 
choi^s and makes the pr e s i deti t ' s r o 1 c tno*re commonly^ 
sport ic and symbolic than significant . ... The American 
college or university is a prototype organized anarchy. It 
does not know what it is doing. Its goals arc either vague 
or inUispute. Its technology is familiar but not. _ 
understood. Its major participants wander in end out of the 
organization (pp^ 2-3). 

the Cohen and Harch thesis did, and does, strike a responsive 
chord for the countless frustrations, uncertainties, and 
uncont^bilables in higher education administration, yet whenever a 
nation, a family, or a university comes upon hard times, more vigorous 
leadership is called for. the extensive academic cutbacki in most U.S. 
colleges and universitiei in the past 10 years have brought forth the 
need foi mbre dynamic leadership, which is quite contrary to the sense 
of presidential helplessness developed by Cohen and March. 

Writing on "Effective Administrators". David Whctten categorized 
administrators as either gatherers or hunters Gatherers forage 
passively in an immediate area for enbugn fooa to satisfy daily needs. 
Hunters aggressively pursue a quarry as far as nc^essery and recognize 
that survival depends upon cunning and strategy." According Co 
Whetten's reBea^ci, the aggressive leadership style of many academic 
edSinistratbrs possesses three d i s t i n^u i ih i Hg char i»c t e r i s t i c s : 

o 12 
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Firsts assi^rtive academic adni t ni s t rator s preserve and 
highlight sources of opportunity within their institutions . 
k . * _ M A secbnd cpmppncnt bf assertive leadership is 
cuitivatir. g a distinctive institutional image. The. third 
cbmponent of assertive leadership toucV.es on the roleof 
charismatic leaderships especially in Lhe strategic planning 
process. Ef f ec t 1 ve _ admi n i s t r a t o r s recognize that in charting 
a. coarsefor an institution, inspiring vision_precedeSi__ 
father than is driven by, quantitative analysis (198^, ppi 
5 . 



And in his widely read book on Acad en ic Strategy , George Keller 
strikes a similar tone: 

Retrenchment, constrictingfihances^newcdmp 

marketing, andrapid changes intheacademicand_de mo graphic 
arear all sp^ll the end of the traditional, unobtrusive 
style pf 'yrganizatibhal leadership bri cainpuscs . • . • 
American Sigher education needs tb transcend the current 
faculty-administration stalemate^ to take. its own management 
mbre seriously, andtocreate new forms of institutional 
decisibn making if it is to cope with and help shape the new 
environment in which it finds itself" (1983, p. 39). 



In suiii^ the climate and circumstances in the U.S.A. now favor the 
kind of vigorous leadership that is a critical factbf in impfbving 
institut ibhal quality. There are some positive signs that the chief 
executive officer styles are reflecting this need more nbw than in the 
past two decades^ but the mills of the Gods grind slowly . • . • 
3 . A^^^ ^D^v^rall plan for enhancing i n s t i tu 1 1 bh a 1 <B^a3r^ tv should be 
developed and cb m 4n u n 1 c a t e d . 

Members of the academic cbmniuhity should have ample opportunity 
tb knbw that ah evaluation effort is underway, with certain 
individuals having some designated responsibilities, with mandates 
and/br charges established to guide the effort, with some reporting 
dates scheduled, with open hearings planned ^ and with action 
recbmmendations anticipated. 



o ^3 



ERIC 



1 1 

Characteristics of i hs t i t u t ibha 1 evaluation have been developed 
by a number of groups and individuals. The Northwest Association of 
Schools and Colleges, for example, determined that the self-study 
process should be flexible and permit different approaches; be attuned 
to current institutional priorities; utilize recent institutional 
research or studies; involve as many people as possible; have 
sufficient breadth and depth to review accountability; foe Us on 
results of the educational prbgrani; use information and results to 
improve the document; yield a concise, readable useful document; and 
ibster bhgbing self-study and planning (1975^ p. 6) • 

The New York State Education Department project on self- 
assessment fbr cblleges and universities suggested the fbll owing steps 
in s e 1 f -a s s e s smen t : 

1. A clear definition of the gbals bf the assessmehti as distinct 
from thegbals bfthe specificarea being.asse ssed, should be_ 
made*. The ma i i f o c u s o f assessment should be on evaluation of 
educat ional q^o slity as measured by goai-ofiented outcomes. 



All persons whb are affected and in terestedinthepro 

under r e v i e w _ s ho u 1 d _ be _ con t i au a 1 1 y made aware of and often 

involved in t he s s s e s sment process. Responsibility for setting 
priorities, designing the assessment, cbllectihg and ahvilyzing 
data^an devaluating and using them should be assigned tb 
appropriately skilled persons. 

A de t e rtrii ha t ion of how we li the gbals bf the specific area 
.issessedare being met should be made. The apprbpriate 
instruments and techniques mustbe selected and administered 
to the constituencies involved (for exatiiple, administrators, 
faculty, students^ graduates, employers, and outside 
groubs]). 



The process of c o 1 1 e c t i n g da t a s h o a 1 d be established in such a 
way that it can continue beybhd the first s e 1 f -a s s e s sme n t as a 
routine function bf the master planning and decisibh-ihakirig 
process • 

Analysis bf data^ reporting bf findings, and recommendations 
foractionshouldbecarefullymonit the persbh(s) 

responsible for the se 1 f -a s s e s smen t • Periodic fbllbw-up bf 
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rccbiniiiendat i ons is essential to determine if any actual 
results have occurred^ 

6. Essential t o e f f ec t i ve _ s e 1 f - a s se s stneh t is the periodic 
evaluation oi the systera itself* _The systein should be 
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CP s t.-e £ f e c t i \ 5 i n both dollars and human time spent to provide 
vital information for deciition making (1 979 , pp, 5-6); 

4 ^ Hod^^^^-^^g^^i— l^^a^s studying and evaluating i h s t l^^^^fc^r ^ a 1 quaV 4^ 

^ould be linked to r e s ou r c^^ l 1 be a t i on an d /or re-allo c ation . 

It has been said that ignorance is a great innovator because we 
never would have undertaken many of bur accomplishments if we knew 
what we were getting into when we started! 

Accepting the earthly wisdbih bf the previous statement does not 
negate the reality of a French saying, which is trar.ilatcd as: "Power 
t? whe^c the money isi and the rest is theatre." One does not have tb 
agree fully with the statement to appreciate its pertinence to 
academic institutions. 

The importance of careful linking academic pi atis /model s to human 
and material resources has been forced upon m^st of us by having tb 
cope with diminishing resources. Still, in the U.S.A. we have a 
tendency to 16osely cbuple. if we couple at all, academic models *nd 
finances. Mbst collegiate administrators believe they do relate models 
and plans to fiHahces but upon closer scrutiny this relatibhship Is 
more often toward the end rather than initially and throughout the 
process. 

5. Institutional evaluation shoul d use W^^t^ data where available 
and purposeful but make no apologies for usir.g sublective d ^^. 

Or. it is better to be ieSerally right than precisely wrong 
(EnthoveS. 1 970 5 . Objective data is i tnpo r t an t , • y e t considerable 
iatlbn exist* in iti availability and quality; the absence of 

15 
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objective data shoaid stiinulate those responsible for i n s t i t a t i ona i 
evaluation to devise their own survey i ns t r uwent s , guidelines, and 
checklists or to use systematically treated judgment as bases for 
decision makihg. The lack of '*hard'* data should not deter careful and 
systematic decision making about important institutional matters. 
Solid bases for decision making can be developed by using whatever 
"hard" data are available along with experiehcei judgment, and cbmTnoh 
sense. Important institutional decisions sometimes are made on 
much less. - 
6 . Those developing models and plans for evalueting institutional 
q ua 1 i ty -s^^HChu^r^ — have ah errbr-expectati bh attitude . 

With this mind-set it is less likely that one will become 
discompbsed and discouraged when things go awry. It may help if one 
tftkes to heart the earthy vrisdom found in two of Murphy's laws: 

- Nbthihg is ever as simple as it seems. 

- if anything can go wrbng» it will. 

Ah accompanying important mind-set is that of retaining the 
mental dexterity necessary for responding to unexpected opportunities. 
One of the mbst dramatic examples of this mental dexterity is the 
quarterback in American football. Competent ones have the ability to 
•'check bff" at the line bf scrimmage; that is, they can call an 
au«iibie play which means changing the upcoming play oh-the-spbt based 
upon how he "reads" the defensive formation that is opposing him. 

How many potentially excellent mbdels and plans can ybu recall 
that were fatally br hear-fatally afflicted with the rigor mortis 
syndrome^ that is^ Ve have an butstahdihg plan that we must follow 
because we must fbllbw it; or with the blinders syndrome^ that is^ one 



can see very wcii straight ahead but is oblivious to everything that 
is going oh elsewhere? 

7 • Institutional evaluation studies should be action-oriented^ v-itH 
p l atvs for fflov t r-ep^ r ts-to - action * 

"Filed and forgotten'^ should hot be ascribed to most 
institutional evaluation efforts, considering the irivestinent of 
thousands of hours and dollars and the importance of keeping the 
institution abreast or ahead of its prdblems» but the history of 
institutional studies does not provide cause for exuberance about 
effective imp 1 enien t a t i o n of recommendations. Today, however, there are 
a number of excellent case studies of instiutional evaluations that 
were action-oriented because the times demanded more rigorous 
approaches to implementing recommehda t ions • 

The administrative charge to an institutional evaluation 
committee should make clear that its work plan is geared toward 
recommendat ions-f or-act ion • This positi<;n from the outset will remind 
those involved in the projectsi and others^ that t'l.'^it efforts are 
less likely to be filed and forgotten. 

8 • A plan for evaluating the evaluation should be included . 

Most evaluation reports make little or no provision for 
evaluating their effectiveness. Such evaluations of evaluations can be 
useful as testimony to the importance of evaluation in future 
imprbvemehti for providing systematic checkpoints of progress, and for 
providing a procedure for orderly modifications based upon subsequent 



findings. 
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Some Probierns in EvgtqgttnR I ng t i n q 1 1 ons 1 Quaiit ^ 
There are many potential prbbleths involved when one begins a 
s'ribus effort to^'ard evaluating institutional quality. And I ain 
certain that you can add several probleitis that you have encountered: 
1 . Poorly defined Roals and objectives can significantly impede the 



If we do hot know where we are going then any road will take us 
there ; _ 

Stateiiiehts of goals and objectives may be flawed in a number b 
ways; some of the common shortcomings are: 

A. Project goals i t\d objectives are hot related to overall 
university goals and objectives; 

B. They are hot cjosely related to the major tasks to be 
accomplished • 

C. They are'^tbb low tb challenge^ br they are too Utopian to 
achieve* 

D. The fbcus is oh activities rather than upbh outputs. 

Ei They are too wordy and full of "academic garbage.** and/or 

they address uhimpbrtaht matters. 
F. They are written more to impress then to implement; 
2 . F a i Iru re tb Rive adequate attention tb the pro c^s^s^s^ i h v o 1 \^e d — i^ 



bringing about improvements can be fatal to the ente r prise . 

In the U.S.A. we are beginning ahbther cycle bf interest in the 
processes of change; A previous cycle was sparked by the Kennedy- 
Jbhhsoh era in the 1960*s when much literature^ research, and 
conversation took place about the "how" dimension. We are moving into 
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this area of interest agaih^ motivated primarily not by growth bat by 
steadv-state or decline. 

•jhere are differences in the meanings of change that are deeply 
rooted in a^cademe. At the risk of oversimplification, it is a 
perceptual problem rooted in differences between acadernic 
disciplinarians and administrators. The latter group foccses more on 
the "bottom line," or on what is produced, while the ac ad j m i c i an s 
focus more on the "how" diTnehsibh^ or the processes involved. 

The management of change in U.S.A. colleges and universities has 
sotiie genuine problems^ and these flaws become more apparent when the 
topic is as complex, important, and sensitive as evaluating 
institutional quality. The title of one chapter in the book by George 
Keller, "Slouching Toward Strategy," captures the problem. Also ah 
ear ier quotation from Cbheh and March's book that spoke of the 
uni\ersity as "organized anarchy". these characterizationi apply even 
tnor< to the way academic, committees go about their work than to the 
academic administration of the inst1!.tution^ 

Typical academic coinib It tees can be defined as gatherings^ usually 
casual, convened late, without agendas, that take minutes but waste 
hours, although often they do hot evr '^ke minutes. bnc faculty 
member said that when he died he hopec at it would be in a faculty 
committee meeting because there the tr^ ion from life to death 

would b' scarcely perceptible! 

3 . A "cargb-cult" ment ality can cause problems . 

Many years ago the natives in the South Seas islands believed 
that a ship laden with gold and precious gems would some day sail intc 
their harbor and bring a happy ending to all their problems. 

o iS 
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Soinc of our approaches to institutional cvaidation bear a 
rcicmblancc in that we latch upon a single answer to complex prbblcBii. 
The latest example is the tendency on part of some American educators 
to look to the iheasureihent of student outcomes as the answer to 
evaluating i n s t i t a t i ona i quality. 

In the inid-1970*K some individuals looked to fiscal indicators as 
t>e answer. 

How nice it would be if institutions could be evaluated so 
s mply. Evaluating i n s t i t a t i ona 1 quality, if it is Anything, is a many 
splendored thing that requires several cbmpbhenL parts that form a 
wlole* which we hc^e will be greater than the sum of the parts. 

Returning for a mbnient tb the outcomes or value-added approach: 
Care tnust be taken that we do not ascribe values to outcomes test 
S'lores that are nbt intended. Oh this point, Peter Eve 11 nbtes: 

Inter pre t ing every score gain that ve observe on a test as an actual 
change in student learhih^l can be a very risky activity (1985^ p. 18). 
And the title of Herbert Spencer's little book in 1S48 is quite 
relevant. It asked: "What Knowledge Is Of Mbst Wbrth?" 
4 ; Making too many changes, too fast can be a problem . 

Generally speaking, academe is nbt knbwh for its rapid changes. 
It has been sa:id that if Ford Motors were a university they would 
still have a department for the Edsel. 

Extensive and cbmp iicated '".hanges take time. in his article 
describing the exemplary value-added program developed at Northeast 
Missouri State University^ Osigweh traces steps in this prbgram back 
to 1973 (1986, pp. 28-33). 
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In a wbrld where instant coffee, fast diets, quick divbrcesi and 
instantaneous worldwide communications are commonplace , is it any 
wonder that we might become impatient with multi-year academic 
improvement efforts? Perhaps we all need to remind ourselves 
occasionally that crash programs usually crash, 

^ • Failur e to gnt^nc^ ^ate problems — c^iv^rbvide shbrt-term euphoria and 
Ibhg-terrh extihctibh . 

While "ignorance can be a great i nhb v a t br ^ t h e other side of the 
coin is that failure to ameliorate weaknesses and problems can be of 
great advantage to the opposition. Su^^cessful athletic coaches spend 
much time on strengthening weaknesses and problem areas to avoid such 
an eventuality. 

In order to better anticipate prbblems. it might be useful to 
consider several reasons why evaluation models and plans fail: 

A. An obvious reasbh. but bne that we prefer to overlook is 
that they are poor plansi bur projects and plans in highe.r 
educatibh rarely fail. Crop failure analyses are common In 
agriculture and autopsies are coTnm'^n in itedicine, but in higher 
education we just write another propbsal. 

B* The model or plan is not vigorous:* y and consistently 
suppbrted at the top. A 1974 study by Davis and Batchelor found 
that only seven percent of college presidents really got fully 
ihvblved in the planning process (1974^ pp. 32-36?. Since the 
study is several years old* one would like to think that some 
imprbvemeht has taken place in the intervening years. 

C. Failure of evaluation programs is possible if the 
enterprise is viewed as the '^reserve bf the institutional 
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planners • 



In s t i t a t 1 onsl researchers and planners are crittcaii 



but their roles arc "incahs" rather than "ends." it is interesting 
to note what is happening in U.S.A. industry. The September 17, 
1984 issue of Bus ihess WeeR magazine titled its front cover and 
feature story, "The New Breed of Strategic planner." Many ma^br 
corporations have cut their planning staffs o the bone, and 
middle man&geinent pe.'sonnel and some line personnel have been 
given planning "know-how" and planning responsibilities. Tiis 
rather dramatic shift is consistent with the growing etiiphas s on 
the entrepreneurial spirit in American business and industry that 
is developed tn Rosabeth Ranter's provocative recent book on Th^ 
Change Masters . 

b. Plans for developing institutional quality are failure- 
prone if tliey do hot carefully consider the iceberg-uhdcr-thc- 
surface^ the detailed managertai system , that is needed to 
develop^ maintain, evaluatei and modify institutional evaluation 
p i an s « , 

E. Another fdrmulafbr failure is the misplaced use of 
evaluation as a smoke screen. A few administrators have sought to 
keep people busy planning and operating so they cannot turn to 
the real problems, on'? of which may be the administrator! 

F. Evaluation plans can be flawed if they do hot have a 
built-in and identified "fixer". This is the English translation 
for a Russian individual that is prominent in the extensively 
planned Soviet system--the "toikash." A procurement specialist^ 
this individual is openly known but bffically a non-person; He or 
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she adds fluidity^ creativity^ and saccess to an often awkward 



planning Bystetti; 



A 



fixer 



II 



in bur collegiate planning process may be senior 



faculty ineinber» inay be soineone in the adir. ihistrative offices^ or 



elsewhere^ who is able to add fluidity, creativity, procedaral 



adjustments, and evaluations to our campUs planning syste:.. «• 



leaving nebulous the responsibility for adjustments in the 
planning processes and syste.ns, we risk significantly weakening 
their impact. 

G. A seventh possible cause for failure is lack of autehtioh 
to a positive and sustained mind-set. Every middle and S'inior 
level collegiate official might replace the IBM slogan of 
"Think," with "Think Strategically." On this point. Ring and 
Cleland wrote: "The abtlity to 'think strategically' in 
comprehensive policy terms is perhaps the least hbm bg e n c b u s 1 y 
distributed persbnal managerial trait" (p. v.). 
Finally, a perspective: Those thoughts about improving 
institut ibhal quality neted to be kept in the larger context of a 
college education. I believe that it must remain robrc than an 
accumulatibn bf credit hours or a pipeline to a position as important 
as are these accomplishments. There is a spirit about a campus that 
sets it apart ^s an unique interlude. Its products and its on-going 
activities account for most of human progress and inventions; its 
stored and alive sources of knowledge contain atid carry forward the 
magnificent heritage and knowledge bf mankind for all to ponder, to 
question^ and to knbw. its traditibns and mores cause frustratibns and 
misunderstandings a&iong its critics; its freedoms provide an eiiential 
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bolwark for the democratic way of life; and iti concern for ideaitts 
and ideas has kindled the hopes and dreans of tnliliens wJro have foiind 
thetiiselves during college days. Most of us are better people becaase 
of this brief period; and this is what counts. 
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APPEWDIX* 



this section will consider these five different approaches 
to Institutional evaluation: 1, ed uca 1 1 ona 1 a ud 1 1 1 ng . 2. externa 
cbhsultant revlewsi 3. self studies for accreditation, A. ielf 
studies for institutional purposes, and 5. state and federal 
reviews • 



1 • l^u« t ional- Ao d l t Ina , The S e c u r i 1 1 e s and Exchange 
Cbminissibh was e s t a b 1 1 she d i n 1 9 3 4 in response to serious 
questions about the conduct and disclosure of business opera- 
tions, in I9i7, the American Institute of Accountant s had 
published a list of s t arid a r d s f 6 r t h e p ? e p a r a t i on b f audits. 
Although the fbrw and nature of the audits vary, the fbllbwihg 
eight basic postulates were developed by the American Accounting 
Association (Mautz and Sharaf, 1961| p. *2): 

1. Financial statements and financial data arevcrifiablc. 

2. There is no necessary conflict of 1 h t r e s be t we en the 
auditor and the managetneht bf the en t e r pr i s e und e r audit, 

3. The flni.ncial & t a t (tmeh t & and b t he r iafbrmation submitted 
for verlficatibn are free from collusive and other 
unusual irregularities. 

i. The existence of a satisfactory system bf internal 

control eliminates the prbbabtlity o f i r r bgu 1 a r i t i e s . 

5. Cbhslsteht applicatibn bf generally accepted principles 
bf accounting results in the fair presentation bf 
financial position and the results bf operations. 

6. In the absence bf clear evidence tb the_ contrary ^ what 
has held true in thepast fbr the enterprise under 
examination will hold true in the future. 

7. When examining financial data fbr the pur pose thereon , 
the auditbr acts exclusively in the capacity bf an 
auditbr. 

8. the professional status of the independent auditbr 
imposes commensurat e profess Ibhal obligations. 

Jbhn Sizer asks whe ther f ive standard s used by the American 
Account ihgAssociat ion can be applied to higher education 
ins t i tut tons • these f 1 ve s t anda r d i are: relevance, vc r 1 f 1 ab 1 1 i t y , 
freedoin from bias, qUant 1 f labi 1 1 t y ^ and economic feasibility (pp. 
75-79). And Pbrter ( 1 9 7 8 5 P r b pb s e d t h a t the test of institutional 
accept ance--arc the indicators fair and ac c e p t a b 1 e - -be added tb 
the five standards. 

As pbihted but byHarclcrbad an Di ckey ( 1 9? 5 , p. 15): "With 
some slight mod i f i c a t i on and changes in terminology, these same 
postulates might very well be applied to the process of 



This material is based upon cited references by the authbf 
in the bibliography plus lecture notes. 
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institutional auditing and accrediting. The v e r i f i a b i i i t y o f 
data^ determination of adequate sampling, and probability theory 
would be characteristic of both business and educational 
institutibni." 



Of the var iou s d i f f er ence s that exist between the businestf 
and educational audits, perhaps the most i mppr t an t _ one r e 1 a t e s t o 
scope. Thfe fiscal audit is only a part of the educational audit, 
but it is tji^ signxficaht ispect for business. Soine individuals, 
par t icularly those with business backgrounds, may relate institu- 
tional healthalmost exclusively to fiscal heaith- Suchclose 
equating should be avoided by institutions of higher education 
because of fundamental differences in purposes and funding 
procedures • 

Harcleroed ( 1 976| p. J 8 ) wr i t e s t ha t e d u c i o na 1 accredita- 
tion can profit in a number of ways from some of the experiences 
of the auditing profession: 



(IJ There might be improvement in the total 
j^rbcess if mpre of the work were to be carried on by 
full-time professionals, without losing the value bf 
having many of the participants serving on a part-time 
basis as at present. (2) Th e r e i s _ a ne cd for a research 
staff i similar to that of the Financial Accounting 
Standard Board, to be wor k i ng on ba s i c principles of 
assessment and procedures to be followed. (3) Sbme fbrm 
of continuity isneeded for the ihstitutibh involved 
and the main members of tbeteam thatwiii work with it 
as a representative of the accreditation body. (4) The 
standards of the regional associations and the 
specialized associations, and their respective 
cbmmis8ibhs» should be similar enough to discount 
claims that great differences invalidate the entire 
process. The business auditing systeihi even with all 
its current problems arid its many aud i t i ng f i rms , ha s 
sufficient cpmparabili ty in i ts standards and their 
application to be quite credible, most of the time. 

Private managerial firms are becbmihg inter es ted in the 
audit approach to higher e d u c a t i on ; and t he first report by 
Newman (1971, p. 76) spoke of the creation of "net regional 
examining universities. "Inconsidering the heed fbrevalua ting 
higher educatibhi Bbweh ( 1979 ^ pp. 1 - 2 1 5 wr o t e t ha t one of the 
nee d s b f h i ghe r _ ed uca t i on i s to find a means of evaluation that 
is g en Q i ne 1 y d i 6 i n t e r e s t e d and yet takes account b f the many 
intangibleeleihehts. Perhaps a hew prbfessibh of independent 
judges of prbductivityandperfbrmahce. should be created to 
evaluate institutions as well as higher education as a whole; 

2i External Cbhsul ^^^i t Studies . A hiimber of ihstitutibhs bf 
higher educatibn as well as state systems have chosen to use 

Q QQ 
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»^ecognl2ed scholars from iirallir un 1 ve r « 1 1 1 * « . . 

various programs. Their report, are briit^^* 
publication «r t-Ki. "re printed in an official 

RecUl ? ' university called The H..„..... . chi.,.» 

The use of e x t er hal r e v i e we r s /eon s u i t .in t i carries th^ 

Co..ittee^.o Evaluate the Oepa^^tent'^^L^^o:: ' UlU'l .0.) 
3f the University of Chicago: -None of us is conf i dent f it ^ 
lltl L^r^''"''; concentrated, can revf"" 1 ^h 

appropriate one s " ' i *p 1 i e s anL fie r po^s i b 1 e Jis'l" 

tional repercussions fro«, 1 1 1 -ad ^ i s^d r ^ «r ^ ' ^ Institu- 

conclosions. The revi IwL . i ' "«ten,ents. or 

those remain^nr^oHli: "i "the^repo^'^'Thr^^^^^ ''"^ 
create more problems than they solve if thei J 

-'^^ «-ong recommenLtiJ^s on :^'ameir; nH^^^e"' 
matters that are not subject to iolution «t the ti»r or 
discusses the wrong probleni and issues. 

O'Conneli and Meeth fl97A « i \ -i ^ zi _ 

Advantages of the internal evaluatbr ar^ th-^ II P ogram^ . taf f . 

familiar with the prorran. and ^I^i^l ^^""^ °- "^e is 

communication within fhrinsti^lio^ ^ ^: -^^""e 1 - - ^ 

familiar with program details and L k ^ ^''^'''" """""''5'' 

evaluation int^ the life of theT ^ integrate the 

internal e.^luator arl that h' or sSe""" ^*"^vantages of the 
^ - - ne or sh<5 may not have the skills 

or'o«Im I --lu-tion. may have a ^e.t.d interest in he 

program, and may be overburdened with other duties. 

follo^r*''* Pottinger (1975). I devised the 

following questions to Assist in choosing and using consultants: 

"''"what^'r: l^"^^^^^^ « reviewer/consultant is needed? 
a. What is the nature of help needed? 
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b. Is the circumstance specif ic enough to articulate clearly 
what if needed in the way of outside assistance? Some 
possible mpcivations include sorting out ambiguous 
problems^ in s t i i 1 1 ng ino 1 1 va t i on , gaining fresh perspec- 

recommending decisions that internal personnel 
prefer riot to make • 

How do ybuchoose theproperconsuitant? 

a. the consultant have expertise in your specific area o 
need? 

b. Several questions might be asked of others who have used 
the consul tan t : 

(1) Did the consultant help you further articulate your 
circumstance without prematurely anticipitihg a 
solution? 

(2) Did the consultant help identify resources and 
approaches within your situation that assisted after 
departure? 

(3) Did you feel more capable of dealing with the problems 
after the cohsu 1 1 an t lef t ? 

(4) Knowing what you know now, would you hire the same 
consultant if you had it to do over again? 

How do ybU most effectively u s e _ c o n s u 1 1 a 1 1 on ? 

a. Make one person clearly responsible and acountable for the 
consultant's work. 

b. Take ample time to define the. problem clearly. 

c. Explore expectations again about what can and cannot be 
done. Ha ve a wr i t t eh r ec or d . 

d. Use the consultant's time effectively. This requires 
carefully developed visitation schedules. 

e. Have a clear understanding of wha t evaluat ions and report s 
are expected or required fron^ the consultant and within 
what time frame. 



3. Self^StUdies for Acc ^^l tat ion . Self-studv for regional 
accreditation has been 1 h ex t is t enc e throughout the twentieth 
century i and it takes place through the six regional accrediting 
associations* The Council on Post secondary Accrcdi ta t ion 5 1 976a , 
p. 5) gives these h i s t b r i c a 1 arid current goa 1 s of a c c r ed i t a t i on : 

- fbsterexcellenctinpostsecondary education through the 
deve i bpmen t of criteria and guidelines for assessing 
educational effectiveness 

- ericburage imprbvemerit through continuous self-study and 
pi anning 

- assure the educational community, the gerieral public, and 
other agencies or organizations that an institution or 
prbgram has both c 1 e a r 1 y d c f i ned a hd a ppr o pr i a t e 
pbjectiveSi ma i n t a i n s_ c bnd i t i on s under which their 
achievement can reasonably be expec ted , appear s in fact 
be accomplishing them substantially^ and can be expected 
to cbritinue tb db so 
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- provide c ouh^ e 1 ^nd a s s i 5 t anc e to estabtiBhed and 
developing institationt and prbgrans 

- encourage the diversity of American pos t secondary 
edac4itioh» arid al lbw_ ini 1 1 1 at ioni to achieve their 
parti ciilar object ives and goals 

- endeavor to protect institutions against encroachments 
that might jeopardize their educational effectiveness or 
academic freedom 

^ r Sei f -Studi e s for In stitutional Purpos e . Hundreds of 
institutional relf-studies uhrelatedto accreditation have been 
initiated in recent yearsi and dozens are underway at any one 
time. The national trend is t b war d i ns t i t a t i ona 1 evaluation with 
greaterrigbr and frequency. Institutional self-stUdy can be an 
important means for se 1 f - i nprov emeh t . The regeneration Of .ne 
University of Kentucky in the m i d- 1 9 60 • i . s par ked by a dynamic 
and experienced hew president, came frcra an extensive self-study, 
which then became the basis for a compt ehen s i v e master plan, 
hayhcw and Ford (i971, p. 125) .hehtioh that •'Stanford University 
shifted its character from that of a s trbng r e g i bna 1 uni vc r s i t y 
appealing to brighti wealthy^ underachieving students to a 
tjniversi ty of i n t e r nat ibna 1 _ s t a t ur e primarily as a result of the 
findings of a self-study^ Stephens College undertook self-study 
when its administration believed that the time had come to 
minimize the traditions of an ear 1 i er <!ra . That self-study was 
iised to loosen the soil of academia so that a new president cbUld 
have a reasonable chance of exercising academic leadership*" 

The result of s e 1 f - s t ud i c s ^ hove ver . have not been evaluated 
•6 any not iceable extent • An exc is the study tit\deftaken by 

.add (1970), which included self-studies done at the University 
of California, Berkeley; University of New Hamp s h i r c ; Un i ver s i t y 
of Toronto; Swarthmbre College; Wesleyan University; Michigan 
State University; Duke Uh i v e r s i t y ; Brown University; Stanford 
University; and two parti< 1 cases : Columbia College and the 
University of California, tos Angeles. The author reached several 
coriclUsiohi based upon firsthand study of these eleven case 
studies as well as upon analysis of other se 1 f - s t ud i e s : 

Unhappi ly , the results of these studies seen to 
lend supp6rt--at least in a negative way--tothe 
eff icacy of pressure politics as a way of bringing 
about change. There is little indication in any of the 
experiences to support the idea that thestndy-ahd- 
report technique is an e f f ec t i ve _ way of gaining 
acceptance of the need for change or creating 
enthusiasm for involvement in developing new policies. 
Where the study-and-report processes were ihtehded 
primarily to challenge the status quo , they largely 
failed to dosbr When the es sent ial object ive was to 
develop the details of a change in the ^ at us ^qM^ after 
the community had already accepted the ne.ed for some 



32 



30 

change, the s t udy- arid-r e po r t processes were much more 
effective ... or where pressures for change from 
outside the faculties were miich in evidence (pp. 
197-198. 200 J - 

The cases indicatein both positive andhegative 
Ways that strongs skillful leadership is virtually 
mandatory for thesuccess of any serious effort at 
educational reform . i . A priwiary task of academic 
leadership is to try tocounterthcpressuresffvorlhg 
the status quo by creating or mslntsinlng an atnosphere 
of receptivity to change {pp. 205-2061. 

On the whole the e du c a t i bna 1 p6 1 i c y changes 
proposed vary considerably 1 n t he 1 r ver. t u r e comcne s s , 
and they often seem to speak 1 nd 1 rec t 1 y - - i f at aii--to 
the deep ma 1 a i se t ha t pr e s eh 1 1 y affects so aiuch of 
American htghercdu cation. Never thelessi if the 
proposals were to be adopted by the institutions ^ 
concerned^ truly consequential changes would be made in 
the educacionfel ppl icies 61 t.iosc institutions. 
Generally speaking, the cases demonstrate that the 
proposals developed tn the studies b jceme le^s venture- 
some or simply disappeared as they passed through the 
various centers of decision itiaking except where ijomc 
form of countervailing power was present [p. 9). 

The self-studies chosen by Ladd were undertrken between 1966 
and 1969, or during the itudeht unrest days. One s o s pc c t s some 
studies were undertaken to al Icvi a t the tense atid pol i t i c i zed 
campUs climate at the time. bressei*s U976 , p. «09>/ words may be 
rei.svant: "When tTie only goal of self-study is the alleviation of 
pressure, the prescrvatiph of accreditation, or th« attainment of 
foundation grant, s uc c e s » or f a i 1 u r e i h attaining t h e goa 1 s o f t en 
ends the self-study." One can surmise that r e s u 1 t s o f s e 1 f - 
studies conducted under these circumstances would >)e minimal, yet 
Ladd^s conclusions bear careful study. 

Research studies have c ompar e d s t ud i e s made of biitside, 
Son it bred internal , and strictly Internal e va I ua ti bh re port s . 
According to Chamber s ( 1 9 79 , p . 32 ) , "these finding show that 
internal evaluation is as reliable and accurate a method of 

current s 1 tuat ion as the bther two* This docs much 
criticism that the seif-study--bh its face--i$ 
iased^ and unreliable." 

sment proc cd are s _ f o r tn s t i t li t i bn s of higher 
have been developed outside of academe. The 
,e Testing (ACT) Program ( 1 9 7 0 ) i n i 1 1 a t ed >i€ 
;elf-Studv Service (ISS) to assist a college or 
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thi^ institution to observe and explore longitudinal trends in 
student development and opinions on canipiis** (p. IJ* Aspects of 
studentservicescoveredin the surveyare faculty advising* 
counseling; financial needs, cxtracurr icolsr advisin^g 
or ientat ion » housing selections* housing advising^ health* and 
remedial. The ISS survey asks _ student s to indicate the degree. of 
import ance _ they a t t ach to . academic _ voc a t i on s and social and non- 
conventional goods. Instruction* school poitcies, and physical 
facilities als-o receive student evaluation. 



the ISS places the student as primary evaluator ; and few 
will disagree with having a significant student input in an 
institutional self-study, provided students answer questions 
within their scope of knowledge and experience. The quality of 
the. facu7^ty*_ academic and institutional leadership* and fis 
management are some e s sen t i a 1 _ component s i n any comprehensive 
s e 1 f - s t ad y s t ode n t s cannot evaluate effectively. Xhe i r op in i on s 
nay be useful, but data on several keyveriables in institutional 
sc. If-stuiv need to be gathered from other constiruencies. 

hlanning (1976)v working with the Research and De ve 1 o pment 
Center for Teacher Education at the University of Texas« has 
developeda troubleshooting checklist (TSC) for higher 
educational settings t o a s s i s t t ho s e cone e r ned vl t h change in 
their assessment of organizational variables predictive of an 
institution's potential forsuccessfullyadopting Innovations. 
TheTSC consists of one hundred Likert-type items grouped in five 
categories: organizationalchange^ otganizational staffs commuhi- 
cations* innovative experience* and s t ud e n t charac t e r i s t ic s i> The 
diagnostic and predictive instrument is designed to aid users in 
es t imat ing _ the effects of particular variables on the adoption/ 
diffusion process. that is* the TSC provides user « with a iDeans 
of systematically organizing descriptive information in a 
predictive way. TK*^ validity of the TSC remains to be determined* 
although data oh reliability are positive. 

the following ten general areas and f or t y-f i vc evaluative 
statements form the framework lor my system of institutional 
evaluation (Miller* 1979): 

Coals and objectives 

1. The goals statement serves as an effective guide for the 

present and future* 
2i bbjectivesreinforcegoals. _ 

3. The i n s t i t u t ion has adequate planning capabilities,^ 

4. Institutional admi ss i ons pc 1 i c i e s and procedures are 
consistent wSthlnstitutibnnl goals ahdbbjectives. 

5. The institution's goals and objectives help it maintain a 
reasonable identity within a statewide system of institu- 
t i ons • 
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Student 4 e^a^4^ ivft 

6; Students give a good rating to their advising and counseling 

systea^ - _ _ 

7. Retention rates are reasonable. ^^^^ ^ ^ ^ 

6i An array of individualized and compensatory learning 

resources isavailable* 
9. The student affairs administration is effective. 

10. Satisfactory progress towrrd learning goals is evident. 

FLaeoltv performance 

11. Current policies and procedures for evaluating individual 
faculty members are satisfactory. 

12. Current instructional imprbVement/fBCulty development 

programs serve their purpose* - _ 

13„ Faculty personnel policies and procedures are considered 
satisfactory. ^ 

Faculty salary scales ard ftirtge benefits ar-; co»npetitive. 
The overall quality 6f the faculty's performance is optimal. 



14 
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Academic programs 

16. The institution has effective policies and procedures for 
developing new programs. _ _ 

17. the institution has ef feet 1 ve :pol i c ie s and procedures for the 
review and evaluation of existing programs. 

18. The general education component is an intellectually 
stimulating sod t n t egr a 1 p a r t 6 f the curficulaio. 

19. The quality and size of the graduate program are consistent 
with institutional goals and objectives. 

20. The library or learnihg resources center provides good 
service to the academic ccramuhifcy. 
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Jnstitutiohal suppo^^^g&j^gj.cej 

The physical ptint and f ac i 1 i t i e s ar e adeqoa t e for the size 
6f the student body and for the hature of the academic 

p r og r am . ; : l j ; , 

The institution has a relevant and current longrrange plan 
for developing and ma in t a i n i hg 1 1 s phy s i C a 3 p 1 an t . 
23. Salaries and other benefits for s u p por t pe r s onne 1 are 

SQfficent to attract and retain compe t en t i nd i v i dua 1 s . 
2A. Systematic procedures are used for evaluating the performence 
of support personneli 

Administrative leadership 

25. The adminis;:i-«tion gives ad e qu a t e a 1 1 en t i on to plinnihg. 

26. The chief campus adtn i n i s t r a t or and h i s or her team have 
effective working relationships with other campus 
administrators. * 
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27. Inst i tutipnai governance pollclei arid procedures allow for 
effective ini t i tu t ibhal management, 

28. The polict es_ and prbcec ures established for administrative 
evaluation and for pr oi e s's i ona 1 development are satisfactory. 

29. the institution hai an effective affirmative action program. 

Fl nan c i a 1^ managemeh t 

50. The tuition and fee structure is coopatible with the institu- 
tion's needs and with the students' capacity to pay. 

3*. T'le ins titutiphhafrtn efficient maiagement syrtem for 
accojnting and financial reporting. 

32. Costs and expenditares are comparable with benchmark 
inst itu t ioni , 

33. the investment portfolio is we 11 managed. 

3^. The instttitibn has an effective system for demonstrating its 
accountability; _ 

jSoverning board 

35. The policies and procedures for conducting board affairs arc 
sat 1 sf actor y . 

36. trustees understand thedifferencesbetweenpolicy formula- 
tion and pol i cy impiementat ion and apply this knowledge. 

37. The governing hoard works effectively with external 
constituencies. 

38; the board contributes positively to improving the 
institution. 

Esiternal rela^ons 

39. T^^ institutibnls act ivi t ief contribute to the quality of 

life in its primary service area. 
4D. the institution has e f f ec t i ve ' r e 1 a t i onsh i p i wi t h the itate 

hi gher _ educa t i oh (coordinating or governing} office. 
^1. the insti tit ion has an effective relationship with the 

federaigovernment* 
^2. The institution is ablift to secure acceptable levels of 

funding fron private sourcej and foundations. 

^4v#^^4M:^l^nBl self - improvement 

43. The institution seeks improvement through innovation and 
experimentation. 

$4. Campus groups have positive attitudes toward self- 
improvement.^ 

i5. The 1 hs t i tu t i ^n ha s established procedur»s for evaluating its 
own effectiveness. 

5. crutg arv^fe^d^r^ reviews > State edacational agencies in 
the U.S.A. have moved aggressively into the area of program 
evaluation since about 1972, largely in an cffcrrt to stem the 



36 



ERLC 



34 



tidc of new graduate prograihs that were be i hg de ye 1 oped without 
careful consideration of t he i r raone t ar y r e qu i r eiiien • » or of iong«* 
range ihirtitutipnalgbAli. New Vb r k , 1 1 1 i no i s . Louisiana, 
Tennessee^ and others were proiainent In there earlier efforts to 
develop stateside progran evaluation efforts. 

-Te«^nessee is one state with a sustained effort, with its 
Per f ormancje Funding Project initiated iti 1 976 by the Tennessee 
Higher Education Cotnmi s s ion « A discussion of the ''bold 
experitncnt" was reported recently by Ted Marchese. This new 
elenent i h t he _ s t a t e f und i ng f o rmu i a is that of having five 
percent incentive funding over and above the regular formula set 
ai5ide for those public universities that c-n vlembnstratc that 
they are making significant progress tbvari 14 numeric goals by 
1990. These 14 gbals are: 

- ACT/ SAT scores of entrants 

- percent of entrants attaining degrees 

- "standardized examination scores" of graduates 

- scores on the National Teachers Examination 

- passrates on "all parts of professional licensing 
examinations on the first attempt" 

- GRE scores bf graduate-school entrants 

- the measured knowledge" of graduate/professional 
f. chool finishers. 

Aid, as a pp 1 i c ab 1 e - by ni s s i on ^ i nc r ea se s a 1 so i n : 

- gift /grant-based rankings for research activities 

- external support for public service programs 
^ job*-!)! acemen t rated by vbcatibnal field 

- t he ma tch be tween vocational offerings and service-* 
area job needs 

- library holdings. 

Plus there is planned tb be _ 

- An end to degree-credit remedial courses 
universities^ and 

- A planned "reduction" in such courses in 
institutibhs (1985^ p. 44). 

Seme r-^cent research has b<:en cov^duc t ed by Robert Barak, 
Deputy Execut ive Secretary of the Iowa Eoard of Regents^ and 
wyself bn the rblebf state level boards bf higher educatibn in 
the review of undergraduate academic program reviews • (I have 
previously mentioned that the role of state level boards of 
higher education in review of graduate academic programs is well 
es tabii shed • ) 

Being the first such national study of the undergraduate 
level) we did hot know quite what to expect. We did find that 29 
states had sbipe fbrm of statewide undergraduate academic program 
review and evaluation. This recent trend in the U.S.A. reflects 



in public 
other 



37 



35 



the continuing ihtereit i h ac c ouh t a b 1 1 1 y » the current einphaats on 
acadeiiilc quality^ and the continuing fiscal constraints on our 
colleges and universities; 

tie inst i t u t ipna 1 peripec t i ve » however, is different; Just 
in view of the number of courses and the number of changes the 
iindergr adua t e prograiD area is much more ^ omp 1 1 ca t ed t han the 
graduate level. And the undergraduate level has been traditional- 
ly where the battles over academic freedom have been fought, 
which titeatis that academics are more sensitive to external 
influences at the undergradua t e 1 e ve 1 . Perhaps pur most 
disappointing finding was that the state level o f f i ce s i n many 
states were making little or ho use of the results of under- 
graduate academic reviews to modify and/or Improve their 
programs * 

The federal government occupies a complex ra^le in higher, 
education. By virtue of the ten th Amendment to the United States 
Bill 6t Right s » education is left t b d i s c r e t a on pi each tftate. 
However^ many fields such as agr ica 1 1 ure , s c i enc e . medic 1 ne» and 
health have been and cbhtihue to be i i gn i f i can t 1 y influenced by 
the federal government* In these areas as well as in a few 
pther«t the federal government has been involved in varying 
degrees of review and evalua t i on . But general ly speaking the 
major r e s pon s 1 b i 1 i t y f or the direction and nature of primary, 
secondary, aud po s t s ecbhdary education is left to the individual 
states * 

the federal gbvcrhmeht has been active in a number of other 
ways that can indirl^ctly or otherwise influence the quality of 
bur colleges and universities. Examples arc federal funds and 
varibfis grahts and 1 bah programs and civil r i gh t s and a f f i rroa t i vc 
action policies and mandates. ,It is no t sar prl s tng , therefore, 
that the U.S. governmehtshbuldi have ah interest in educational _ 
qualityi as illustrated by the highly publicized book on A^^tion 
at Ris k. Federal concern, however, is d i f f e r en t _ f r pm f ede r a 
control or exc e s s i ve i nf 1 uehce c Ins t itut ional and s tate-levt 1 
officials are willing tovbrk coo pe ra t i ve ly wi t h the federal 
gbvernmeht in achieving mu t u a 1 1 y d e s i r ab 1 e goa 1 s of better 
quality control through cooperation and coordination. 
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Twentieth Special Topic Workshop of the I^fiE Prograjnme. We Workshop will 
provide participants with an opportunity to examine how institutional 
evaluation is carried out in a nuiaber of OECD countries^ discuss the various 
problems which arise in this context and exchange experience on progress to 
date. 
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THE ROLE OF EV.UUATION! IN TFE MANAGBEiNTT OF, INSTITUTIONS 

BE- -EIJGHB? EiXJGATIQiN? 



In many OECD countries^ institutions of higher education have been 
preoccupied in recent years With coming to. grips_With.proble:rs posed in the 
context of economic constraint- The activities of the I>WE Programme have 
reflected the desire of institutions to respond effectively to these 
conditions. _It is generally recognised that many responses in such conditions 
tend to be of a short-term natureand a consensus is emerging thatsteps will 
need to be taken in order to_ bring about a restructuring of individual 
institutions which will ensure their financial viability and academic quality 
over a longer period of ecdndmic staghatidn.: A.precdhditidn for such 
restructuring is the development of Eraineworks for institutional evaluation. 
Such frajTieworks foi* evaluation will be the focal point of this workshop. 

To some extent, forms b£ evaluation are alwaysjpresentih academic 
institutions. There exist_in every country individual academics, departments, 
research institutes and centres of learning which are renowned nationally and 
internationally. The mechanisms which make it possible td; assess the quality 
of individuals and groups at the level of specific disciplihes:are based upon 
a long tradition of peer evaluation by means of scientific publications, 
national and international conferences-, colloquia, etc. The problem of 
evaluating institutions of higher education for purposes of improving the 
quality of the institution by means of managerial initiatives has dnly 
recently.begun to be looked at and this Workshop will explore the current 
status of this subject. 



PURPOSE 

The purpose oT this Workshop will be td examine how institutional 
evaluation is carried out in a number of QEBd countries and discUss the role 
which evaluatidn can play in the management of institutions 6F_higher_ 
education. Methods of anstituUdnal self-assessment as well as evaluation by 
external bodies will be covered and approaches^ developed and used in North 
America and Europe will be analysed andcompared, with a view toward assessing 
the extent to which generalised conclusions may be drawi from these 
experiences* 



PAR-Tiei PANTS 



The Workshop should be of particular interest td senior staff in 
institutions of higher education who are involved in policy formulation and 
decision-making* This could include executive heads (presidents, rectors^ 
directors* vice-chancellors), heads of administration (registrars, kanzlers, 
vice-presidents for administratidn) ^ deans and other, senior staff , The 
Workshop should ais5 be of interest to representatives of national^agen^ 
directly concernve with the evaluation of institutions of higher education. 
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CdNTENTL 



There is a growing heed to develop sour 1, practical approaches to 
institutional evaluation in the field of higher education. This is in part 
due to increased demands for institutibrial accountability which are being made 
by society. But it is also the result of a recognition by institutions 
themselves that without systematic methods of self-appraisal it is extremely: 
difficult to manage a complex organisation such as a univers.ity, especially in 
a period of economic stagnation. There is a need toidentify both centres of 
actual and potential excellence as well as areas which_Call for improvement, 
Continuing assessments and appraisals are part of the managerial task of 
ensuring better-run^ high-quality institutions. 



For purposes of this V/orfcshop^ institutional evaluation is meant to 
cover the development of criteria and procedures for appraising all aspects of 
the institution^ with a view toward assisting management to ascertain the 
extent to which the institution is going where it wants to go. Among the 
aspects of the institution which mahagv^n.ent may wish uo evaluaie are: goals 
and objectives, student learning^ academic staff performance, academic 
prograinmeSi institutional support services, administrative leadership, 
financi^l.maiiagement^ governance^ _^e and institutional 

self-improvement. The opening paper will provide a framework within which the 
various cases to be presented during thecourse of the Workshop can be^ 
discussed. To obtain a broad apc^triim of cUrreht practice, examples of _ 
institutional evaluation from France, thf.Netherlaids^ the_United Kingdom and 
the United States will be presented. Due to differences among countries, it 
is not expected that each experience presented will cover all of the aspects 
of institutional evaluation c;ted above. Rather,^^ they will hi|hll|ht_ those 
procedures currently in use to assess institutional perfonnahce and enable 
participants to discuss the state-of-the-art of such procedures both for the 
external evaluation of institutions as well as institutional sislf-appraisal. 



Among the questions which will be considered during the course of the 
Workshop are: How can roechanisms for undertaking institutional evaluations be 
set Up? JVhat are the roles of national, regional or other bodies in 
evaluating institutions? What are the main components of systems of 
self-evaluation? How can the results of institutional evaluations be used for 
improving the management of the institution at lall levels? Hdw can problems 
of internal tensions which may arise as part of evaluation exercises be 
reduced? What are the_likely future developments in the field of 
institutional evaluation? 

_: The Workshop will begin with an overview of approaches to and problems 
of evaluating institutional quality. The remainder of the first day will be 
devoted to a presentation and discussion of experiences of institutional 
evaluation in the United States and France. Additional country experiences 
from the_Uni ted Kingdom, _ the Netherlands and the United States will be 
presented and discussed on the second day. Tne last morning will begin with a 
presentationby the Workshop rapporteur which will be followed by a panel 
discussion of likely future developments in the field of institutional 
evaluation. Oh Wednesday afternooh^^ a Tutorial will be held at which 
participants will have an opportunity, to examine in. greater technical detail 
several evaluation procedures currently in Use at the University of Tennessee, 
Kiioxvillei 

^1 
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PR£LIMI?^ARY SCHEDULE 

^Registration will begin at 9.50 a. m.^dn Monday, 12th May 1986; Morning 
sessions are scheduled from 10 a.m. to 1 p.m. and afternoon sessions frocn__^__ 
ip.in. to 6 p.m. The Workshop will end at 12 noon on Wednesday, :14th Hay 1986. 
The special tutorial on the Use of student dutcbme iafbrraation in assessing 
and improving academic programmes will take place frora 2.30 p.m. to 5 p.m. oh 
Wednesday ^ 14th May 1986. 



Monday Marhi^g - A IJ.S. Perspective on Evaluation 

1. Presentation of the Workshop by the Secretariat. 

2. Introductory paper: '/Evaluating Institutional Quality: Some Ways and 
Some Problems'* - Richard I. Miller, Ohio University, United States. 

3. Break 

4-. Presentation of Case 1 - "Quality Assessment at the University or - 
Tennessee Knoxville'' - Trudy W. Banta, Homer Fisher and G.W. Minkel, 
United States 

5; Discussion 



Monday afternoon - Recent Developments in Institutional Evaluation in France 
1. 'The Role and Function of the National Evaluation Gommittee for 





Universities" - 


Andre Staropoli^ Coraite National d' Evaluation, France 


2. 


Discussion 




3. 


Break 




4. 


Presentation of 


ease 2 - To be announced 


5. 


Discussion. 






[jgher Education in 



the United Kingdom 



1. "The Council for National Academic /awards : Peer Review and 
Partnership" • Alan Hibbert, CNAA, United Kingdom 



2. 



Discussion 



Break 



Presentation df_Case 3 - "An Institutional Perspective on External 
Validation and Review" - John Stoddart ^ Sheffield City Polytechnic, 
United Kingdom 
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Tuesday afternoon - Institation a l S elf -Apprai sal 

1. Presentation of Case 4 - ''Improvement of the Quality of Education 
through the Use of AMED (Analysis Model for Education at the 
Disciplinary Level)^' - Pieter Drenth, Vrije Universiteit Amsterdajrii 
Netherlands 

2. Discussion 
5; Break 

4. Presentation of Case 5 - ''the Value Added Program at Northeast Missouri 
State University" - Charles J. McClain, United States 

5; Discussion 



h eine:>Qa/ ..lorning Cjix^er.t Status ani Futuie De >..lopmcnts 

i. Summary by the Workshop rapporteur - Michel Bernard, Universite de 
Nantes i France 

2i Panel: **Likely Future Developments in the Field of Institutional 
Evaluation" (Speakers to be announced). 

3; Discussion 

4. Official closing. 



Wednesday afternoon - J^rto rial : Use of Student Outcome Infonn ax ion in- 
Assessing and Improving Academic Prograjtimes 

During the Workshop, the approach developed by the University of 
Tennessee Knoxville (UTK) will be__presented. _ On the occasion of the visit by 
the teajn of : representatives of UTK responsible fotjthe development and 
implementation of this approach^ a special tutorial has_been organised at 
which it will be possible for Workshop participants to examine in more - 
technical detail the methods developed there as well as how the results are 
used in academi e programme revi ew and : i n planning for programme. improvement . 
Among the topics to be covered a re_: (i) the_use of the College Outcome 
Pleasures Project (CG?»iP) examination to assess the "value adaed**_by the 
collegiate experience in general education and_in each student's majbrj 
(ii) the use of comprehensive examination results and student^ drop-puts and 
alumni opinions to_evaluate and improve curricula and (iii) the design of 
instruments and methodologies for surveying currently enrolled students and 
alumni to assessztheir opinions of the_quality of academic programmes _ and _ 
services. In 1984, _ the bniversity of Tennessee Khoxville received the annual 
award of the National Council on Measurement _in_Educat ion for "an outstanding 
example of the application of educational measurement technology". The 
Tutorial will be conductedby Professor Trudy W. Bant a af the University s 
Learning Research Center, Executive. Vice-Chancellor Horrier Fisher and 
Vice-Provost e,K. Mihkel. It will be.chaired by Professor Richard 1. Milieri 
Q author of the Jossey-Bass book The As s essme n t o f College Performance, 
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_^ :__ _ _Partieipatiori at the Workshop is open to representatives of D-HE member 
in<;titutions as well as to government officials and representatives of 
national organisations responsible for hi|her_educatioh. Persons wishing to 
attend are asked to complete and return the enclosed Cdnf irmatiori of 
Attendance_form to the Secretariat before 10th April 1986 , Persons requiring 
hotel reservations are asked to return their forms as soon as possible. 



pffiTjeiPANT eemiSirriQ?^ 



Speakisrs appearing oh the Workshop prog rafnme have Been invited to 
document their presentations and their papers will be made available to 
Workshop participants in English and French. In. addition, any part icpant 
wishing to prepare a gaper tin English or French) on the subject of the 
Workshop may do so. Such papers will be considered for inclusion in a future 
^f!:':e_of the in-temat ional .hxmizl of Institutional Manage:::ent ir. Higher 
Education, ° 

^If you_wish to prepare such a paper, you should indicate this on your 
application forro. Upon receipt of your fonti, informatfon pertaining to the 
preparation of Journal art icles: Will_be sent to you. You w4ll also be 
inforiTjed about the expected number of participants in order that you may bring 
a sufficient number of copies to the Workshop if you wish to distribute your 
paper at that time. 
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